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Introduction 

n an international scale, higher education re¬ 
form is having a profound impact on organiza¬ 
tions and institutions, where there are now mandates 
and requirements to implement explicit learning out¬ 
comes and assessment policies for all undergraduate 
curricula (Bresciani, 2006; Hubball & Burt, 2004; 
Hubball & Burt, 2007). Program-level learning out¬ 
comes are a central component of learning-centred 
curricula and inform students what they can expect 
to achieve from a program of study so that they may 
organise their time and efforts, and prepare for as¬ 
sessments. They also connect segments of a cur¬ 
riculum, thus enhancing transferability of student 
learning, communicate curriculum/program goals 
in a meaningful way to a broader community, help 
to determine the extent to which learning has been 
accomplished, and guide faculty and administrators 
(within resource constraints), in part, to determine 
program(s) of study, course objectives, appropriate 
learning experiences, and assessment and program 
evaluation strategies (Barab & Duffy, 2000; Hubball 
& Gold, 2007). This paper highlights program-level 


assessment of learning outcomes in an 8-month Fac¬ 
ulty Certificate Program (UBC-FCP) on the scholar¬ 
ship of teaching and learning (SoTL) at the Univer¬ 
sity of British Columbia (UBC). 

Program Context 

The UBC-FCP on SoTL began in 1998. UBC, Cana¬ 
dian, and International faculty members participated 
in the program to investigate critical curriculum and 
pedagogical issues within their department (e.g., cur¬ 
riculum re-design, evaluation of curricula, PBL, staff 
development, web-based learning), while some fac¬ 
ulty members embark on this program for personal 
reasons to improve course design, teaching, learning 
strategies, and assessment practices. The UBC-FCP 
has graduated nine multidisciplinary faculty cohorts 
since 1999. UBC President Stephen Toope awards 
the Certificates to graduating faculty at Green Col¬ 
lege in May following completion of the program. 
Faculty graduates include national and international 
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teaching award winners, CRC’s, full professors, ten¬ 
ured faculty, tenure-tack faculty and instructors from 
across UBC campus (Hubball & Poole, 2004; Hub- 
ball, Pratt, & Collins, 2005; Hubball & Burt, 2006; 
Hubball & Albon, 2007) 

The aim of the UBC-FCP is to enhance SoTL 
in multidisciplinary settings. Essentially, SoTL is an 
approach to academic work that integrates research, 
teaching, and learning. The literature differentiates 
between scholarly approaches to teaching and learn¬ 
ing, and the scholarship of teaching and learning 
(Richlin, 2001; Kreber, 2001; McKinney, 2004). 
In the context of the UBC-FCP, the following op¬ 
erational definition for SoTL guides curriculum, and 
teaching and learning processes. Scholarly approach¬ 
es to teaching and learning refer to on-going profes¬ 
sional development, reflection, and initiation of posi¬ 
tive changes to curricula and/or pedagogical practices. 
Taking this concept to the next level of rigour, SoTL 
refers to the dissemination of practice-driven curricula 
and/or pedagogical research in peer-review contexts. 

Action Research and Program-level 
Assessment of Learning Outcomes 

Action research (AR) methodology is at the very 
heart of SoTL. In this context, AR was employed to 
strengthen the underlying theory/rationale for learn¬ 
ing experiences within a program, and gain authentic 
data on which to (cyclically) reflect on the effective¬ 
ness of processes and outcomes (Altrichter, Psch, & 
Somekh, 1993; Hubball & Clarke, 2004; Hubball 
& Levy, 2004; Wolfe, Hill, & Evers, 2006). Action 
research methodology was employed to address four 
specific research questions pertaining to program- 
level assessment of learning outcomes: 

1 What are the critical factors when developing 
program-level learning outcomes? 

2 To what extent are learning outcomes reflected 
in program learning experiences? 

3 When and how do faculty members demon¬ 
strate learning outcomes in this context? 

4 What are the overall reflections for implementa¬ 


tion of program-level learning outcomes in this 
context? 

The following data were gathered to address the 
above research questions: a critical review of faculty 
members’ SoTL portfolios; program literature sourc¬ 
es from the website and curriculum materials; cohort 
members’ program evaluation feedback pertaining to 
the quality of program, teaching and learning experi¬ 
ences; focus group interviews with cohort members’ 
video footage of research presentations; and instruc¬ 
tors’ debrief and review of lecture notes. 

Results 

1) What are the critical factors when 
developing program-level learning outcomes? 

The following framework and critical elements guid¬ 
ed the development and implementation of program- 
level learning outcomes: 

Learning context strategies. This refers to a com¬ 
prehensive needs assessment which was conducted 
by consulting with various sources (e.g., a focus 
group of UBC faculty members from various aca¬ 
demic ranks and disciplines who were committed to 
the SoTL and willing to participate in a pilot-pro- 
gram) to guide the development of program-level 
learning outcomes. 

Planning strategies. This refers to the logistics of 
program scheduling, as well as the development of 
explicit core program-level learning outcomes; i.e., 
in the context of the SoTL, UBC-FCP faculty will 
demonstrate: 1) the acquisition, application and in¬ 
tegration of SoTL knowledge; 2) research skills, in¬ 
cluding the ability to define problems and access, re¬ 
trieve and evaluate SoTL information; 3) a critically 
reflective practice and problem-solving abilities with 
respect to SoTL; 4) responsible use of ethical prin¬ 
ciples; and 5) effective leadership, communication, 
and interpersonal skills (Table 1 shows how specific 
learning outcomes relate to UBC-FCP learning ex¬ 
periences). 
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Table i 



Assessment strategies. This refers to the implemen¬ 
tation of a wide range of assessment methods (e.g., 
self-reflection, peer-feedback, instructor feedback, 
and external peer-review) to assess program-level 
learning outcomes. 

Programming strategies. Depending on Prior Learn¬ 
ing Assessment (PLA), each faculty participant in 
this FCP context follows an individual learning plan 
which combines theory, practice, and critical reflec¬ 
tion. Guided by the program-level learning outcomes, 
faculty participants have on-line weekly assignments, 
as well as a meeting each month, and at different loca¬ 
tions on campus to engage in a wide range of indepen¬ 
dent and/or collaborative learning experiences. 


2) To what extent are learning outcomes 
reflected in program learning experiences? 

A faculty member’s SoTL portfolio is a compilation 
assignment used to demonstrate how program-level 
learning outcomes (listed above) feature in a wide 
range of authentic learning experiences. Table 1 cap¬ 
tures the range of UBC-FCP learning experiences 
that are employed to address the five UBC-FCP pro¬ 
gram-level learning outcomes. 

3) When and how do faculty members 
demonstrate learning outcomes in this 
context? 

For example, feedback on a faculty member’s teach- 
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ing dossier is provided at four key incremental stages 
throughout the 8-month program. Further, an ex¬ 
ternal review takes place at the 40 and 60 percent 
(informal formative feedback provided toward each 
SoTL assignment within the faculty member’s port¬ 
folio) stages of the program, as well as during the final 
month (formal summative process) of the 8-month 
program. In the context of a wide range of UBC- 
FCP learning experiences, this not only allows for 
progressive development of each assignment within 
the SoTL portfolio, it also staggers the completion 
deadlines for overall SoTL assignments and demon¬ 
stration of program-level learning outcomes. 

4) What are the overall reflections for 
implementation of program-level learning 
outcomes in this context? 

The flexibility, quantity, and quality of overall pro¬ 
gram-level learning outcomes are critical issues when 
shaping teaching, learning, and assessment experi¬ 
ences. In many cases, learning experiences integrated 
multiple and overlapping program-level learning out¬ 
comes. This was not viewed as excessive, redundant, 
or repetitive, but rather an indication that program- 
level learning outcomes can be interpreted in different 
ways and demonstrated in many authentic ways. Too 
many program-level learning outcomes, however, can 
create surface learning rather than deep and meaning¬ 
ful learning. Not all faculty members were aware of 
the implicit connections between program-level learn¬ 
ing outcomes and program experiences. Furthermore, 
a number of faculty members were unsure about the 
priority of program-level learning outcomes over 
personal and alternative outcomes. Thus, the devel¬ 
opment and implementation of program-level learn¬ 
ing outcomes is not an exact science. Furthermore, 
programs involve complex, multifaceted, and contex¬ 
tually-bound program realities and are influenced by 
many people at various levels of implementation. 

Conclusion 

Universities and academic units face considerable 
challenges in developing, adopting, and implement¬ 


ing program-level learning outcomes within univer¬ 
sity curricula. Despite well-coordinated, innovative, 
and strategic attempts to implement program-level 
learning outcomes, they are unlikely to fully occur 
as intended due to the highly complex world of cur¬ 
riculum practice. 

This article provides a flexible and iterative 
framework to examine whether and how program- 
level learning outcomes are implemented in a faculty 
certificate program. Data from this study suggest that 
educational leadership, learning context, planning, 
programming and assessment processes were critical 
factors in the development and implementation of 
program-level learning outcomes. 
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